Method, storage medium and electronic device for detecting vehicle crashes

ABSTRACT

The present disclosure relates to method, storage medium and electronic device for detecting vehicle crashes. The method comprises: acquiring state information of a target vehicle; and determining an event type of the target vehicle according to the state information and a trained convolutional neural network, the event type being any of the following types: a crash event, a near crash event and a baseline event. The event type of the vehicle is determined using the trained convolutional neural network in the present disclosure, so that the accuracy is high; and near crash events can be detected, thus, when a near crash event is detected, the driver can be further alerted or an evading operation is directly performed on the vehicle, so that the safety is improved and the safety of the driver and passengers is guaranteed.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of Chinese Application No.201710672705.1, filed Aug. 8, 2017, which is incorporated by referenceas if fully set forth.

FIELD OF THE INVENTION

The present disclosure relates to the field of vehicle data processing,and specifically, relates to method, apparatus, storage medium,electronic device and vehicle for detecting vehicle crashes.

BACKGROUND OF THE INVENTION

Automatic detection of vehicle crashes is beneficial to timely notifyingcrash accidents to relevant personnel and organizations, includingfirst-aid personnel, family members, team principals and insurancecompanies. On the other hand, timely detection of crash accidents isalso beneficial to investigating the accidents.

In some relevant technologies, vehicle crashes are automaticallydetected directly using crash detection hardware sensors. In some otherrelevant technologies, operational data of vehicles are acquired usingvehicle-mounted sensors or mobile sensors, and feature values arecalculated via the methods of integration, difference and the likeaccording to the sensor data. Then, thresholds are calculated via thesefeature values to determine whether crashes happen.

SUMMARY OF THE INVENTION

In order to solve the problems in relevant technologies, the presentdisclosure is aimed at providing method, apparatus, electronic deviceand vehicle for detecting vehicle crashes.

In a first aspect, the present disclosure provides a method fordetecting vehicle crashes, including:

acquiring state information of a target vehicle; and

determining an event type of the target vehicle according to the stateinformation and a trained convolutional neural network, the event typebeing any of the following types: a crash event, a near crash event anda baseline event.

In a second aspect, the present disclosure provides an apparatus fordetecting vehicle crashes, including:

an acquisition module, used for acquiring state information of a targetvehicle; and

a determination module, used for determining an event type of the targetvehicle according to the state information and a trained convolutionalneural network, the event type being any of the following types: a crashevent, a near crash event and a baseline event.

In a third aspect, the present disclosure provides a computer readablestorage medium, storing a computer program which, when executed by aprocessor, performs the steps of said method.

In a fourth aspect, the present disclosure provides an electronicdevice, including:

the computer readable storage medium in said third aspect; and

one or more processors, used for executing the program in the computerreadable storage medium.

In a fifth aspect, the present disclosure provides a vehicle, including:

the computer readable storage medium in said third aspect; and

one or more processors, used for executing the program in the computerreadable storage medium.

In said technical solutions, the event type of the vehicle is determinedusing the trained convolutional neural network, so that the accuracy ishigh; and near crash events can be detected, thus, when a near crashevent is detected, the driver can be further alerted or an evadingoperation (braking, abrupt turning, or the like) can be directlyperformed on the vehicle, so that safety is improved and the safety ofthe driver and passengers is guaranteed.

Other features and advantages of the present disclosure will bedescribed in detail in the following specific embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used for providing further understandingon the present disclosure, constituting a part of the specification, andinterpreting the present disclosure together with the following specificembodiments, rather than limiting the present disclosure. In thedrawings:

FIG. 1 is a schematic diagram of a vehicle of an embodiment of thepresent disclosure;

FIG. 2 is a schematic flow diagram of a method for detecting vehiclecrashes of an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a vehicle of another embodiment of thepresent disclosure;

FIG. 4 is a schematic flow diagram of marking an event type via imagerecognition in an embodiment of the present disclosure;

FIG. 5 is a schematic flow diagram of training a convolutional neuralnetwork in an embodiment of the present disclosure;

FIG. 6 is a schematic flow diagram of preprocessing a training sample inan embodiment of the present disclosure;

FIG. 7 is a schematic flow diagram of augmenting time series data from asensor in an embodiment of the present disclosure;

FIG. 8 is a schematic flow diagram of merging time series data based ontimestamps in an embodiment of the present disclosure;

FIG. 9 is an input schematic diagram of a convolutional neural networkto be trained in an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of a convolutional neural network of anembodiment of the present disclosure;

FIG. 11 is schematic diagram of a convolutional neural network adoptedin an embodiment of the present disclosure;

FIG. 12 is a schematic flow diagram of testing the trained convolutionalneural network in an embodiment of the present disclosure;

FIG. 13 is a block diagram of an apparatus for detecting vehicle crashesof an embodiment of the present disclosure; and

FIG. 14 is a block diagram of an electronic device shown according to anexemplary embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The specific embodiments of the present disclosure will be described indetail below in combination with the accompanying drawings. It should beunderstood that the specific embodiments described herein are merelyused for illustrating and interpreting the present disclosure, ratherthan limiting the present disclosure.

Refer to FIG. 1, which is a schematic diagram of a vehicle of anembodiment of the present disclosure. Sensors 10 may be a GPS (GlobalPositioning System), an accelerometer, a gyroscope and the like arrangedin a vehicle. In some embodiments, the sensors 10 may also be arrangedin an electronic device which is placed in a vehicle, therefore, thestates of the sensors in the electronic device can reflect stateinformation of the vehicle.

State information of the vehicle can be acquired via the sensors 10. Thespeed of the vehicle, as well as the longitude, latitude, height, courseand the like at different time points can be acquired via the GPS. Theaccelerometer can acquire accelerations in X, Y and Z directions atdifferent time points. The gyroscope can acquire angular speeds of thevehicle at different time points, including angular speeds in the Xdirection, angular speeds in the Y direction and angular speeds in the Zdirection.

Refer to FIG. 2, which is a schematic flow diagram of a method fordetecting vehicle crashes of an embodiment of the present disclosure.The method includes the steps as follow.

In step S21, state information of a target vehicle is acquired.

In step S22, an event type of the target vehicle is determined accordingto the state information and a trained convolutional neural network.

In the embodiment of the present disclosure, the event type of thevehicle is determined according to the state information of the vehicleacquired in real time and the trained convolutional neural network.Wherein, the event type is any of the following types: a crash event, anear crash event and a baseline event.

The crash event refers to that the vehicle is in crash contact with amoving or motionless obstacle (e.g., other vehicle, building, etc.), andthe original running speed of the vehicle is obviously transferred ordisappears. When the crash event happens, one or more of the followingsituations generally happen: the safety airbag deployment of the vehiclecollapses; a driver, a pedestrian or a bicycle rider is injured; thevehicle turns over; a very large speed change or acceleration changeoccurs; traction of other vehicle is needed; and property loss iscaused. The resulting personal injury needs the help of doctors.Besides, a crash with a large animal, a crash with an sign posts and thelike also belong to crash events.

The near crash event is any event in which the current vehicle needs toquickly evade to avoid a crash. The vehicle does not contact any movingor fixed obstacle. The word “evade” herein means to control turning,braking or deceleration of the vehicle or a combination thereof, therebyavoiding the potential crash. Evading is quick, i.e., the time that thedriver of the vehicle makes a response is short. In the near crashevent, the distance between the vehicle and the obstacle is controlledwithin a certain range (e.g., 0.1 to 0.5 meter).

The baseline event involves a normal driving behavior without crash ornear crash.

In the method for detecting vehicle crashes of the embodiment of thepresent disclosure, the event type of the vehicle is determined usingthe trained convolutional neural network, so that the accuracy is high;and near crash events can be detected, thus, when a near crash event isdetected, the driver can be further alerted or an evading operation(braking, abrupt turning, or the like) is directly performed on thevehicle, so that safety is improved and the safety of the driver andpassengers is guaranteed.

The convolutional neural network in the abovementioned step S22 isobtained by training, and the training process of the convolutionalneural network will be introduced below.

The data acquired by the sensors is time series data, including stateinformation of the vehicle recorded by the sensors according to time,i.e., state information with timestamps. In one embodiment, the stateinformation of the vehicle includes: speed, acceleration in the Xdirection, acceleration in the Y direction, acceleration in the Zdirection, angular speed in the X direction, angular speed in the Ydirection and angular speed in the Z direction.

The training sample adopted in the training phase is time series datawith event type tags. Wherein, the time series data corresponding toeach time point has an event type tag.

In the embodiment of the present disclosure, the time series data withevent type tags can be acquired in any of the following modes.

Mode I: an event type corresponding to the vehicle image is determinedby means of image recognition according to a vehicle image acquired byan image acquisition device, and marking of the event type is performedon the time series data corresponding to the vehicle image.

Referring to FIG. 3, the vehicle includes an image acquisition device 30arranged in the vehicle. The image acquisition device 30 may be acamera, a dashboard camera or the like, and is used for acquiringvehicle images. There may be a plurality of image acquisition device 30,which are respectively arranged at the front, back, left and right partsof the body of the vehicle. The acquired vehicle images include an imagein front of the vehicle, an image in back of the vehicle, an image onthe side of the vehicle, etc. The image in front of the vehicle canreflect a vehicle event of the vehicle and a front obstacle. The imagein back of the vehicle can reflect a vehicle event of the vehicle and aback obstacle. The image on the side of the vehicle can reflect avehicle event of the vehicle and a side obstacle.

Refer to FIG. 4, which is a schematic flow diagram of marking an eventtype via image recognition in an embodiment of the present disclosure.

In step S41, image recognition is performed according to a vehicle imageacquired by an image acquisition device.

The process of image recognition on the vehicle image may includepreprocessing of the vehicle image, recognition on the body of thevehicle and the obstacle around the vehicle, etc.

In step S42, an event type is determined according to the result ofimage recognition.

An event type is determined according to the vehicle image acquired bythe image acquisition device 30, for example, the image acquired andpreprocessed by the image acquisition device 30 can be matched with astandard image, then the similarity between the acquired vehicle imageand the standard image is determined according to the matching result,and whether the vehicle undergoes crash or near crash with the obstacleis determined according to the similarity. The standard image may be avehicle image not undergoing crash or near crash among the acquiredvehicle images. The standard image may also be a vehicle imageundergoing crash or near crash among the acquired vehicle images. Thevehicle images acquired by the image acquisition device 301 may be theones of the vehicle in the front, back, left and right directions, andthus, in determining the standard image, vehicle images corresponding todifferent directions can be respectively determined according todifferent directions.

In step S43, the time series data corresponding to the vehicle image ismarked according to the determined event type.

Based on mode I, event type marking is performed on the time series datavia image recognition, so that the time series data with event type tagscan be used as the training sample.

Mode II: a vehicle event corresponding to the vehicle image isdetermined by means of artificial recognition according to the vehicleimage acquired by the image acquisition device, and thus, marking of anevent type is performed on the time series data corresponding to thevehicle image to obtain time series data with an event type tag.

Mode III: time series data with event type tags is acquired from arelevant database and used as a training sample. For example, timeseries data with event type tags can be acquired from a database of anatural driving research project.

When mode I and mode II described above are adopted, the image acquiredby the image acquisition device 301 is synchronous with the time seriesdata acquired by the sensor. The time when the event corresponding tothe time series data occurs can be accurately positioned via the imageacquired by the image acquisition device 301 and the time series dataacquired by the sensor, thereby realizing accurate event type marking onthe time series data.

Refer to FIG. 5, which is a schematic flow diagram of training aconvolutional neural network in an embodiment of the present disclosure.

In step S51, a training sample is acquired.

In step S52, a convolutional neural network is trained according to thetraining sample and a training termination condition.

In step S53, when the training is terminated, parameter information ofthe convolutional neural network is acquired, the parameter informationat least including: weights of a convolution layer, biases of theconvolution layer, weights of a pooling layer, biases of the poolinglayer, weights of a fully connected layer, biases of the fully connectedlayer, number of convolution layers, size of the convolution kernel ofeach convolution layer, number of pooling layers, size of each poolinglayer, number of fully connected layers and size of each fully connectedlayer.

In step S54, a convolutional neural network is constructed according tothe parameter information. The convolutional neural network is used forpredicting an event type of a vehicle.

In an embodiment of the present disclosure, in step S51, after timeseries data with event type tags is acquired in any of the modesdescribed above as the training sample, it further includespreprocessing on data of the training sample.

Refer to FIG. 6, which is a schematic flow diagram of preprocessing atraining sample in an embodiment of the present disclosure.

In step S61, time series data with an event type tag from at least onesensor is acquired.

In step S62, the time series data with event type tags from differentsensors is merged based on timestamps.

In step S63, the merged time series data with event type tags isdetermined as the training sample.

Referring to FIG. 7, in an embodiment of the present disclosure, timeseries data from sensors can be augmented to increase the data volume ofa training sample for training, step S62 described above includes:

In step S71, among the time series data with event type tags fromdifferent sensors, the time series data of the same event type issegmented into multiple pieces of time series data based on a minimumtime window corresponding to the event and a preset time window movingamount.

For the time series data with event type tags from different sensors,the time series data of the same event type can be recognized accordingto the event type tags. Thus, the time series data of the same eventtype can be segmented according to the minimum time window correspondingto the event of the event type and the preset time window moving amountto increase the trained data volume. For example, for a crash event, thetime length of the minimum time window may be 6 seconds, the preset timewindow moving amount may be 0.5 second, then from a certain time pointt, the time series data within (t+6) seconds is the first segment ofdata, the time series data within (t+6+0.5) seconds is the secondsegment of data, and so on, till the termination condition is satisfied.The termination condition may refer to that the event types of the timeseries data are no longer same after moving according to the preset timewindow moving amount, e.g., when moving N times, the time series datawithin [(t+6+0.5)*N] seconds is no longer crash time. The terminationcondition may also refer to that the number of movements reaches a setvalue, e.g., the set value of the number of movements may be M, andsegmentation is stopped when moving to [(t+6+0.5)*M].

Thus, the time series data of the same event type can be segmented intomultiple segments to increase the data volume, the event type of eachsegment is same. When the convolutional neural network is trained later,each segment of time series data obtained by segmentation can be used asan input.

In step S72, the segmented time series data from different sensors ismerged based on timestamps.

As mentioned above, the state information of the vehicle may includevehicle speed, acceleration, angular speed, etc., and the data may beacquired by different sensors, so the acquired time series data withevent type tags is data from different sensors. In some embodiments,when different sensors acquire and record data, hardware faults orsignal transmission faults may happen, the data acquisition frequenciesof different sensors may also be different, thus, if the timestamps ofthe time series data from different sensors are different, missingvalues are filled into the time series data via a linear interpolationcompensation method.

Referring to FIG. 8, in an embodiment of the present disclosure, whenthe time series data is merged based on the timestamps and thetimestamps need to be unified, the step of merging the time series databased on the timestamps includes:

In step S81, when the timestamps of the time series data from differentsensors are different, linear interpolation compensation is performed onthe time series data with a low sampling frequency.

For example, the sampling frequency of the data from the sensor 1 is 10Hz, and the sampling frequency of the data from the sensor 2 is 100 Hz,so that the timestamps are different. The data with the samplingfrequency of 10 Hz is interpolated to the high frequency of 100 Hzfirst, so that the data from the sensor 1 and the data from the sensor 2are both 100 Hz and have the same timestamp.

The time series data from different sensors in this step may be the timeseries data acquired in step S61 described above, or the time seriesdata segmented in step S71 described above.

In step S82, the time series data after linear interpolationcompensation is merged to obtain time series data to be sampled.

After step S81, the time series data has the same timestamp, and can bemerged. In the embodiment of the present disclosure, merging enables thetime series data from different sensors at the same time point to bealigned.

Referring to table 1 below, each row of data in table 1 is data obtainedafter merging the time series data from different sensors at the sametime point.

In step S83, the time series data sampled from the time series data tobe sampled based on a preset sampling frequency and the correspondingevent type tags thereof are used as the training sample.

After step S82 described above, the sampling frequencies of the timeseries data among the time series data to be sampled are unified, e.g.,unified to a higher sampling frequency 100 Hz. In step S83, the timeseries data serving as a training sample can be acquired from the mergedtime series data to be sampled based on a preset sampling frequency(e.g., 10 Hz). It should be understood that the preset samplingfrequency in step S83 can be set according to the data volume requiredfor training.

As mentioned above, after the training sample is preprocessed, step S52described above can be executed, i.e., the convolutional neural networkis trained according to the training sample and a preset number ofiterations. The convolutional neural network to be trained herein hasinitial parameter information, which is continually adjusted in thetraining process.

Refer to table 1, which shows a piece of time series data serving as aninput in the training sample. The time series data serving as the inputof the convolutional neural network can be obtained by segmentationaccording to the method shown in FIG. 7 or directly obtained accordingto step S62.

TABLE 1 Channel Channel Channel Channel Channel 1 Channel 2 Channel 3 45 6 7 5.872 2.371 −7.482 −6.067 0.003 0.004 0.008 5.891 2.348 −7.409−5.913 0.011 0.012 0.019 5.91 2.436 −7.441 −6.174 0.003 0.011 0.01 5.9292.362 −7.426 −6.199 0.002 0.005 0.004 5.948 2.237 −7.356 −5.882 0.0110.014 0.017 5.966 2.347 −7.385 −5.963 0.007 0.015 0.015 5.985 2.456−7.481 −6.277 −0.002 0.005 0.002 6.004 2.274 −7.449 −6.044 0.007 0.0090.013 6.023 2.36 −7.49 −5.919 0.009 0.016 0.016 6.042 2.523 −7.538−6.234 −0.001 0.009 0.002 6.061 2.388 −7.486 −6.2 0.004 0.005 0.005 . .. 6.645 2.315 −7.305 −5.953 0.009 0.016 0.013 6.664 2.325 −7.37 −5.935−0.003 0.007 0 6.682 2.546 −7.424 −6.088 −0.001 0.021 −0.013 6.701 2.313−7.541 −6.195 −0.084 0.044 −0.109 6.72 2.567 −7.295 −6.359 −0.616 0.181−0.661 6.739 3.357 −5.967 −7.196 −0.841 −0.649 −0.975 6.758 2.517 −4.837−7.988 −0.675 −0.962 −0.95

The time series data shown in table 1 includes 7 signal channels, thetime length is 6 seconds, the sampling frequency is 10 Hz, and the datasegment is thus a 60×7 two-dimensional array. The 7 signal channelsrespectively correspond to different state information of the vehicle:speed, orthogonal acceleration in the x orthogonal direction, orthogonalacceleration in the y orthogonal direction, orthogonal acceleration inthe z orthogonal direction, angular speed in the x orthogonal direction,angular speed in the y orthogonal direction and angular speed in the zorthogonal direction.

Referring to FIG. 9, in an embodiment, the input of the convolutionalneural network to be trained is multiple pieces of time series data withthe height of 1, the width of 60, and 7 channels.

Referring to FIG. 10, the convolutional neural network in an embodimentof the present disclosure includes an input layer 101, a convolutionlayer 102, a pooling layer 103, a fully connected layer 104 and anoutput layer 105.

The convolution layer 102 is used for extracting the feature of eachchannel of the input time series data. A group of weights for extractingthese features form a convolution kernel. The convolution kernel moveson each channel with a stride, and is convolved with data to obtainfeature mapping. A bias coefficient is added to each convolution result,and calculation is performed through an activation function to obtain anoutput result of the convolution layer.

In the embodiment of the present disclosure, the channels share onechannel multiplier.

The pooling layer 103 is used for performing sub sampling on datasegments, thereby reducing the data processing amount and simultaneouslyreserving useful information. The pooling layer is located behind theconvolution layer, and samples, on the feature mapping of theconvolution layer, a point (e.g., maximum sampling, mean sampling,random sampling, etc.) in an area having a fixed size as an input of thenext layer.

The fully connected layer 104 is connected with the pooling layer 103,and connects all neurons obtained by the pooling layer to each neuron ofthe fully connected layer respectively. Each neuron of the fullyconnected layer is connected with the neurons of all output feature mapsof previous layer, and all the obtained feature maps are arranged in theform of column vectors via an activation function to obtain an output.

Each output of the fully connected layer 104 can be regarded as a sumobtained by adding a bias b to the product of each node of previouslayer and a weight W.

The activation function for the fully connected layer 104 may be ahyperbolic tangent function, e.g., a Tan h function.

The output layer 105 multiplies a column vector output by the fullyconnected layer with a weight matrix, and then adds a bias term andgenerates a column vector via an activation function. In the embodimentof the present disclosure, a K-dimensional column vector is generated ina softmax form, and the value of each column vector element representsthe probability of that type. As there are three event types to bedetermined in the embodiment of the present disclosure, then K may be 3,i.e., a 3-dimensional column vector, respectively representing theprobability of a crash event, a near crash event or a base line event.The event having the maximum probability is a final prediction result.

In the embodiment of the present disclosure, the output of each layer issubjected to rectified linear unit (ReLU) non-linearity, e.g.,non-linearity via an activation function. The activation function may bea Sigmoid function, a Relu function, a Tan h function, etc.

When the convolutional neural network is trained, the forwardpropagation phase is as follows:

The time series data, with event type tags, of the training sample isinput to the convolution layer 102 via the input layer 101. The timeseries data progressively transformed by the convolution layer 102, thepooling layer 103 and the fully connected layer 104 is transmitted tothe output layer 105.

The backward propagation phase is as follows: the weights and biases ofthe convolution layer 102, the pooling layer 103 and the fully connectedlayer 104 are adjusted according to the output result of the outputlayer 105 and the event type tag corresponding to each piece of timeseries data, so that the error between the output result of the outputlayer 105 and the event type tag corresponding to each piece of timeseries data is minimum.

When the training termination condition is met, the weights and biasesof the convolution layer 102, the pooling layer 103 and the fullyconnected layer 104, as well as the number and size of each layer, etc.,are respectively acquired.

In an embodiment of the present disclosure, when the weights and thebiases are optimized in each iteration process of the training process,a stochastic gradient descent method is adopted, and the learning ratecan be set to be 0.0001. In one embodiment, the learning rate is halvedafter every iteration to improve the training efficiency. All the timeseries data in the training sample is submitted sequentially in oneiteration.

In an embodiment of the present disclosure, the training terminationcondition may be as follows: the number of iterations reaches a maximumone, or the error absolute values of judgment probabilities of the eventtypes corresponding to all the time series data in the training sampleare smaller than a preset threshold.

In an embodiment of the present disclosure, in order to improve theability of generalization of the neural network, a preset number ofneurons among the neurons of the fully connected layer are discarded ateach iteration. For example, the preset number may be 50% of the totalnumber of neurons at the previous iteration.

When the training is terminated, parameter information of theconvolutional neural network is acquired: corresponding weights andbiases of a convolution layer, corresponding weights and biases of apooling layer, corresponding weights and biases of a fully connectedlayer, number of convolution layers, size of the convolution kernel ofeach convolution layer, number of pooling layers, size of each poolinglayer, number of fully connected layers, size of each fully connectedlayer and an activation function adopted in each layer.

Referring to FIG. 11, in an embodiment, the trained convolutional neuralnetwork for predicting vehicle crashes includes two convolution layers,one pooling layer, one fully connected layer and one output layer. Thesize of the convolution kernel of the first convolution layer is 1×6,the stride is 1, and the channel multiplier of each channel is 8; thesize of the convolution kernel of the second convolution layer is 1×4,the stride is 1, and the channel multiplier of each channel is 10; thesize of the sliding window of the pooling layer is 1×20, and the strideis 4; and at the first iteration, the fully connected layer includes 60neurons. The output of the output layer is a probability of each eventtype. The activation functions for the first convolution layer and thesecond convolution layer may be a ReLU function.

Test of the Convolutional Neural Network

In order to ensure the prediction accuracy of the convolutional neuralnetwork, the trained convolutional neural network is tested by using atest sample in this embodiment of the present disclosure.

Referring to FIG. 12, in step S120, a test sample is acquired. The testsample includes state information of a vehicle to be tested and an eventtype tag corresponding to the state information. In an embodiment, thetest sample is selected from the training sample, e.g., 30% of sample insaid training sample can be used as a test sample to test the predictionaccuracy of the trained convolutional neural network.

In step S121, the state information of the vehicle to be tested is inputto a convolutional neural network constructed with the parameterinformation to acquire an event type of the vehicle to be tested.

In step S122, when the acquired event type of the vehicle to be testedis not accordant with the event type tag, the convolutional neuralnetwork is retrained according to the training sample to update theparameter information.

When the trained convolutional neural network is tested using the testsample, it can be set that the trained convolutional neural network isused for detecting vehicle crashes when the prediction accuracy reachesa certain value, e.g., 95%. As shown in FIG. 2 described above, theacquired state information of the target vehicle is input to the trainedconvolutional neural network to detect whether the vehicle undergoes acrash event or a near crash event.

It should be understood that the acquired state information of thetarget vehicle is data from different sensors recorded according totime, and the state information recorded according to time can form timeseries data. In an embodiment, the time series data is preprocessed andthen input to the trained convolutional neural network. Thepreprocessing can be performed in the manner shown in FIG. 8 describedabove, i.e., the time series data having non-unified timestamps isprocessed and merged.

The time series data of the target vehicle input to the trainedconvolutional neural network and the time series data in the trainingsample are same in height, width and number of channels.

When the target vehicle undergoes a near crash event, alertinginformation can be output to alert a driver to timely perform an evadingoperation (e.g., braking, abrupt turning, or the like) or directlyperform an evading operation on the vehicle.

When the target vehicle undergoes a crash event, alarm information canbe output, e.g., an alarm is emitted for rescue. In an embodiment, theinformation and position of the target vehicle and the like are sent toa corresponding contact according to the contact information set by theowner of the target vehicle. The contacts may be a first-aid person, afamily member, an insurance company, etc.

In the embodiment of the present disclosure, crash or near crash of thevehicle is detected and recognized via the convolutional neural network,and parameters of the convolutional neural network are determined bylearning, so that the accuracy of crash recognition is high. Besides, anear crash event can also be detected to assist the driver in a timelymanner performing an evading operation, e.g., braking, abrupt turning,etc.

Correspondingly, referring to FIG. 13, an embodiment of the presentdisclosure further provides a apparatus for detecting vehicle crashes,the apparatus 1300 including:

an acquisition module 1301, used for acquiring state information of atarget vehicle; and

a determination module 1302, used for determining an event type of thetarget vehicle according to the state information and a trainedconvolutional neural network, the event type being any of the followingtypes: a crash event, a near crash event and a baseline event.

In an embodiment, the apparatus 1300 further includes:

a sample acquisition module, used for acquiring a training sample, thetraining sample including: multiple pieces of time series data and eventtype tags corresponding to each piece of time series data, wherein eachpiece of time series data includes state information of the vehiclerecorded by at least one sensor according to time;

a training module, used for training a convolutional neural network tobe trained according to the training sample and a training terminationcondition;

a parameter information acquisition module, used for, when the trainingis terminated, acquiring parameter information of the convolutionalneural network to be trained, the parameter information at leastincluding: weights of a convolution layer, biases of the convolutionlayer, weights of a pooling layer, biases of the pooling layer, weightsof a fully connected layer, biases of the fully connected layer, numberof convolution layers, size of the convolution kernel of eachconvolution layer, number of pooling layers, size of each pooling layer,number of fully connected layers and size of each fully connected layer;and

a convolutional neural network construction module, used forconstructing the convolutional neural network according to the parameterinformation.

In an embodiment, the sample acquisition module 1303 includes:

a time series data acquisition sub-module, used for acquiring timeseries data with an event type tag from at least one sensor;

a merging sub-module, used for merging the time series data with eventtype tags from different sensors based on timestamps; and

a training sample determination sub-module, used for determining themerged time series data with event type tags as the training sample.

In an embodiment, the merging sub-module is used for, among the timeseries data with event type tags from different sensors, segmenting thetime series data of the same event type into multiple pieces of timeseries data based on a minimum time window corresponding to the eventand a preset time window moving amount; and merging the segmented timeseries data from different sensors based on timestamps.

In an embodiment, the merging sub-module is used for, when thetimestamps of the time series data from different sensors are different,performing linear interpolation compensation on the time series datawith a low sampling frequency; and merging the time series data afterlinear interpolation compensation to obtain time series data to besampled; and

the training sample determination sub-module is used for determiningtime series data sampled from the time series data to be sampled basedon a preset sampling frequency and the corresponding event type tagsthereof as the training sample.

In an embodiment, the apparatus 1300 further includes:

a test sample acquisition module, used for acquiring a test sample, thetest sample including state information of a vehicle to be tested and anevent type tag corresponding to the state information;

a test module, used for inputting the state information of the vehicleto be tested into a convolutional neural network constructed with theparameter information to acquire an event type of the vehicle to betested; and

an update module, used for, when the acquired event type of the vehicleto be tested is not accordant with the event type tag, retraining theconvolutional neural network according to the training sample to updatethe parameter information.

Regarding the apparatus in the above embodiments, the specific mode ofoperation executed by each module has been described in detail in theembodiment about the method, and thus is not elaborated herein.

Correspondingly, the present disclosure further provides a computerreadable storage medium, storing a computer program which, when executedby a processor, performs the steps of said method for detecting vehiclecrashes.

Correspondingly, the present disclosure further provides an electronicdevice, including: said computer readable storage medium; and one ormore processors, used for executing the program in the computer readablestorage medium.

FIG. 14 is a block diagram of an electronic device 1400 shown accordingto an exemplary embodiment. As shown in FIG. 14, the electronic device1400 may include a processor 1401, a memory 1402, a multimedia component1403, an input/output (I/O) interface 1404, a communication component1405 and one or more said sensors. The electronic device 1400 may be asmart phone provided with hardware such as a GPS, an accelerometer, agyroscope and the like, and when being placed in a target vehicle, theelectronic device can acquire data reflecting state information of thevehicle. On the other hand, the electronic device can also communicatewith the target vehicle via the communication component 1405 to acquirethe state information of the vehicle in real time. Besides, theelectronic device 1400 can store a trained convolutional neural networkinto the memory 1402 or train a convolutional neural network accordingto the method above to obtain a trained convolutional neural network,thus, the trained convolutional neural network is called via theprocessor 1401, and the event type of the vehicle is determinedaccording to the acquired state information of the vehicle.

The processor 1401 is used for controlling overall operation of theelectronic device 1400 to accomplish all of or part of the steps of saidmethod for detecting vehicle crashes. The memory 1402 is used forstoring various types of data to support the operation in the electronicdevice 1400, and the data, for example, may include instructions for anyapplication or method operated on the electronic device 1400 and datarelated to applications, e.g., contact data, received and transmittedmessages, pictures, audio, video, etc. The memory 1402 may beimplemented by any type of volatile or non-volatile storage device or acombination thereof, e.g., a static random access memory (SRAM), anelectrically erasable programmable read-only memory (EEPROM), anerasable programmable read-only memory (EPROM), a programmable read-onlymemory (PROM), a read-only memory (ROM), a magnetic memory, a flashmemory, a magnetic disc or an optical disc. The multimedia component1403 may include a screen and an audio component. The screen may be atouch screen, and the audio component is used for outputting and/orinputting audio signals. For example, the audio component may include amicrophone, which is used for receiving external audio signals. Thereceived audio signals may be further stored in the memory 1402 ortransmitted via the communication component 1405. The audio componentfurther includes at least one loudspeaker for outputting audio signals.The I/O interface 1404 provides an interface between the processor 1401and other interface module, and the other interface module may be akeyboard, a mouse, buttons or the like. These buttons may be virtualbuttons or physical buttons. The communication component 1405 is usedfor wired or wireless communication between the electronic device 1400and other devices. Wireless communication refers to, for example, Wi-Fi,Bluetooth, near field communication (NFC), 2G, 3G, 4G or 5G or acombination thereof, and thus the corresponding communication component1405 may include a Wi-Fi module, a Bluetooth module, an NFC module, a 2Gmodule, a 3G module, a 4G module or a 5G module.

In an exemplary embodiment, the electronic device 1400 may beimplemented by one or more application specific integrated circuits(ASICs), digital signal processors (DSPs), digital signal processingdevices (DSPDs), programmable logic devices (PLDs), field programmablegate arrays (FPGAs), controllers, microcontrollers, microprocessors orother electronic components, and is used for executing said method fordetecting vehicle crashes.

Correspondingly, an embodiment of the present disclosure furtherprovides a vehicle, including: said computer readable storage medium;and one or more processors, used for executing the program in thecomputer readable storage medium.

Preferred embodiments of the present disclosure are described in detailabove in combination with the accompanying drawings, but the presentdisclosure is not limited to the specific details in the aboveembodiments. Many simple modifications may be made to the technicalsolutions of the present disclosure within the technical conception ofthe present disclosure, and all these simple modifications fall into theprotection scope of the present disclosure.

In addition, it should be noted that the specific technical featuresdescribed in said specific embodiments may be combined in anyappropriate mode without conflicts. In order to avoid unnecessaryrepetition, various possible combinations would not be additionallydescribed in the present disclosure.

Moreover, various different embodiments of the present disclosure mayalso be combined randomly, and the combinations should be regarded ascontents disclosed by the present disclosure as long as they do not goagainst the thought of the present disclosure.

1. A method for detecting vehicle crashes, comprising: acquiring stateinformation of a target vehicle; and determining an event type of thetarget vehicle according to the state information and a trainedconvolutional neural network, the event type being any of the followingtypes: a crash event, a near crash event and a baseline event.
 2. Themethod of claim 1, further comprising: acquiring a training sample,wherein the training sample comprises: multiple pieces of time seriesdata and event type tags corresponding to each piece of time seriesdata, wherein each piece of time series data comprises state informationof the vehicle recorded by at least one sensor according to time;training a convolutional neural network according to the training sampleand a training termination condition; when the training is terminated,acquiring parameter information of the convolutional neural network tobe trained, wherein the parameter information at least comprises:weights of a convolution layer, biases of the convolution layer, weightsof a pooling layer, biases of the pooling layer, weights of a fullyconnected layer, biases of the fully connected layer, number ofconvolution layers, size of the convolution kernel of each convolutionlayer, number of pooling layers, size of each pooling layer, number offully connected layers and size of each fully connected layer; andconstructing the convolutional neural network according to the parameterinformation.
 3. The method of claim 2, wherein the step of acquiring atraining sample comprises: acquiring time series data with an event typetag from at least one sensor; merging the time series data with eventtype tags from different sensors based on timestamps; and determiningthe merged time series data with event type tags as the training sample.4. The method of claim 3, wherein the step of merging the time seriesdata with event type tags from different sensors based on timestampscomprises: among the time series data with event type tags fromdifferent sensors, segmenting the time series data of the same eventtype into multiple pieces of time series data based on a minimum timewindow corresponding to the event and a preset time window movingamount; and merging the segmented time series data from differentsensors based on timestamps.
 5. The method of claim 4, wherein the stepof merging based on timestamps comprises: when the timestamps of thetime series data from different sensors are different, performing linearinterpolation on the time series data with a low sampling frequency; andmerging the time series data after linear interpolation to obtain timeseries data to be sampled; the step of determining the merged timeseries data with event type tags as the training sample comprises:determining time series data sampled from the time series data to besampled at a preset sampling frequency and the corresponding event typetags thereof as the training sample.
 6. The method of claim 2, furthercomprising: discarding a preset number of neurons in the fully connectedlayer at each iteration.
 7. The method of claim 2, further comprising:acquiring a test sample, wherein the test sample comprises stateinformation of a vehicle to be tested and an event type tagcorresponding to the state information; inputting the state informationof the vehicle to be tested into a convolutional neural networkconstructed with the parameter information to acquire an event type ofthe vehicle to be tested; and when the acquired event type of thevehicle to be tested is not accordant with the event type tag,retraining the convolutional neural network according to the trainingsample to update the parameter information.
 8. The method of claim 1,wherein the state information of the vehicle comprises: speed,acceleration in the X direction, acceleration in the Y direction,acceleration in the Z direction, angular speed in the X direction,angular speed in the Y direction and angular speed in the Z direction.9. The method of claim 1, wherein the step of determining an event typeof the target vehicle according to the state information and a trainedconvolutional neural network, comprises: preprocessing the stateinformation; and determining an event type of the target vehicleaccording to the preprocessed state information and a trainedconvolutional neural network.
 10. The method of claim 9, wherein thestate information is the time series data from different sensorsrecorded according to time; the step of preprocessing the stateinformation comprises: merging the time series data from differentsensors recorded according to time, based on timestamps.
 11. The methodof claim 1, further comprising: when the event type of the targetvehicle is the near crash event, outputting alerting information; andwhen the event type of the target vehicle is the crash event, outputtingalarm information.
 12. A computer readable storage medium, storing acomputer program which, when executed by a processor, performs the stepsof the method of claim
 1. 13. An electronic device, comprising: thecomputer readable storage medium of claim 12; and one or moreprocessors, used for executing the program in the computer readablestorage medium.